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June  30,  2000 

The  Honorable  Fred  Thompson 

Chairman,  Committee  on  Governmental  Affairs 

United  States  Senate 

The  Honorable  Joseph  I.  Lieberman 

Ranking  Member,  Committee  on  Governmental  Affairs 

United  States  Senate 

Subject:  Observations  on  the  National  Aeronautics  and  Space  Administration’s 
Fiscal  Year  1999  Performance  Report  and  Fiscal  Year  2001  Performance  Plan 

As  you  requested,  we  have  reviewed  the  24  Chief  Financial  Officers  (CFO)  Act 
agencies’  fiscal  year  1999  performance  reports  and  fiscal  year  2001  performance 
plans  required  by  the  Government  Performance  and  Results  Act  of  1993  (GPRA).  In 
essence,  under  GPRA  annual  performance  plans  are  to  establish  performance  goals 
and  measures  covering  a  given  fiscal  year  and  provide  the  direct  linkage  between  an 
agency’s  longer-term  goals  and  day-to-day  activities.  Annual  performance  reports  are 
to  subsequently  report  on  the  degree  to  which  those  performance  goals  were  met. 
This  letter  contains  two  enclosures  responding  to  your  request  concerning  key 
program  outcomes  and  major  management  challenges  at  the  National  Aeronautics 
and  Space  Administration  (NASA) .  Enclosure  I  to  this  letter  provides  our 
observations  on  NASA’s  fiscal  year  1999  actual  and  fiscal  year  2001  planned 
performance  for  the  key  outcomes  that  you  identified  as  important  mission  areas  for 
the  agency.  These  key  outcomes  are  (1)  expand  scientific  knowledge  of  the  Earth 
system,  (2)  deploy  and  operate  the  International  Space  Station  safely  and  cost- 
effectively,  and  (3)  expand  the  commercial  development  of  space.  Enclosure  II  lists 
the  major  management  challenges  facing  the  agency  that  we  and  NASA’s  Inspector 
General  identified,  how  their  fiscal  year  1999  performance  report  discusses  the 
progress  the  agency  made  in  resolving  these  challenges,  and  the  applicable  goals  and 
measures  in  the  fiscal  year  2001  performance  plan. 
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Results  in  Brief 

NASA  made  progress  in  meeting  its  fiscal  year  1999  performance  targets  for  the  three 
key  agency  outcomes  included  in  our  assessment.1  Reported  fiscal  year  1999 
performance  objectives  and  targets  were  generally  objective  and  measurable. 
However,  we  also  identified  several  weaknesses.  For  example,  in  its  fiscal  year  1999 
performance  report,  NASA  indicates  that  it,  along  with  other  federal  agencies  that  are 
engaged  in  science  and  technology,  has  difficulty  in  quantifying  outcomes  and 
relating  outcomes  to  spending  because  of  the  long-term  character  of  its  objectives. 
The  fiscal  year  2001  performance  plan  still  relies  heavily  on  output  measures.  We 
believe  that  the  continued  use  of  output  measures  burdens  NASA  by  requiring  it  to 
continuously  demonstrate  the  linkages  between  program  efforts  and  results  and  to 
make  improvements  needed  to  strengthen  such  linkages. 

NASA  also  does  not  provide  reasonable  assurance  in  its  fiscal  year  1999  performance 
report  that  the  performance  information  used  is  credible.  Discussion  of  data 
credibility  is  limited  to  stating  that  each  of  the  program  and  project  managers  are 
fully  accountable  for  the  accuracy  of  the  performance  information.  Moreover,  the 
fiscal  year  2001  performance  plan  still  does  not  include  an  explicit  discussion  of 
procedures  for  verifying  and  validating  performance  data  and  does  not  address 
possible  data  limitation  issues  and  problems.  In  general,  the  fiscal  year  2001 
performance  plan  still  does  not  (1)  provide  a  clear  rationale  for  how  information 
technology-related  strategies  and  programs  will  contribute  to  achieving  NASA’s  goals 
or  (2)  show  any  allocation  of  information  technology-related  dollars  and  personnel  to 
performance  goals. 

Of  further  concern  is  NASA’s  intended  use  of  performance  indicators  starting  with 
the  fiscal  year  2001  performance  plan.  Our  specific  concern  is  not  related  to  the 
indicators  themselves  but  to  how  NASA  plans  to  use  them  in  assessing  whether  a 
performance  target  has  or  has  not  been  met.  A  NASA  official  indicated  that  a  target 
could  be  assessed  as  having  been  met  even  if  some  of  the  indicators  supporting  the 
target  were  not  achieved.  We  believe  that  this  approach  requires  NASA  to  provide 
convincing  evidence  that  a  desired  outcome  was  met  despite  the  nonachievement  of 
one  or  more  of  the  supporting  indicators.  We  have  the  following  additional 
observations  on  NASA’s  fiscal  year  1999  performance  report  and  fiscal  year  2001 
performance  plan: 

•  For  the  three  key  outcomes  we  evaluated,  the  performance  report  includes  all 
targets  established  in  the  fiscal  year  1999  performance  plan.  Although  the 
performance  information  articulates  the  degree  to  which  the  targets  were  or  were 
not  achieved,  some  additional  explanation  is  needed.  For  example,  for  the  key 
outcome  of  expanding  scientific  knowledge  of  the  Earth  system,  the  performance 


'  NASA's  fiscal  year  1999  performance  plan  and  report  identify  an  objective  closely  related  to  one  of 
the  key  outcomes  you  identified — deploying  and  operating  the  International  Space  Station  safely  and 
cost-effectively.  The  objective  is  stated  as  deploying  and  operating  the  space  station  for  research, 
engineering,  and  exploration  activities.  We  based  our  assessment  on  that  objective. 
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assessments  provide  a  reasonable  explanation  for  all  of  the  targets  that  were  not 
met  except  for  one;  in  that  case,  a  clear  explanation  for  why  the  target  was  not 
met  was  not  provided.  As  a  result,  the  impact  of  not  having  data  from  Global 
Positioning  System  receivers  to  test  improved  algorithms  for  measuring  the 
atmosphere’s  temperature  on  this  key  outcome  cannot  be  determined  and 
corrective  actions  cannot  be  assessed. 

•  The  fiscal  year  1999  performance  report  includes  an  independent  evaluation 
completed  by  the  NASA  Advisory  Council.  According  to  the  performance  report, 
the  Council  provided  a  qualitative  evaluation  of  NASA’s  fiscal  year  1999 
performance,  going  beyond  the  question  of  whether  NASA  met  specific 
performance  targets  in  the  fiscal  1999  performance  plan.  For  example,  the 
Council  identified  the  need  to  develop  a  crew  return  vehicle  for  the  International 
Space  Station — a  suggestion  that  enhances  confidence  in  NASA’s  ability  to  meet 
the  key  outcome  of  deploying  and  operating  the  space  station  safely  and  cost- 
effectively.  We  also  believe  that  NASA  should  establish  targets  for  safety,  cost- 
control,  and  risk  mitigation  measures  to  strengthen  this  related  key  outcome. 

•  NASA’s  Advisory  Council  also  commented  on  a  key  performance  target  related  to 
the  outcome  of  expanding  the  commercial  development  of  space.  That 
performance  target  was  to  continue  the  X-33  vehicle  assembly  in  preparation  for 
flight-testing.  The  Council  commented  that  while  assembly  of  the  X-33  vehicle 
was  on  track  near  the  end  of  fiscal  year  1999,  enormous  challenges  remained. 
While  NASA  met  its  fiscal  year  1999  target  of  continuing  the  X-33  vehicle 
assembly,  a  hydrogen  tank  failure  in  November  1999  exacerbated  the  challenges 
this  program  faces.  NASA  states  that  a  performance  indicator  for  the  X-33 
program  cannot  be  included  in  the  fiscal  year  2001  performance  plan  until  the 
hydrogen  tank  delamination  investigation  and  program  impact  assessment  are 
complete.  We  believe  that  close  monitoring  of  the  X-33  vehicle  assembly  is 
critical.  As  a  key  technology  demonstrator  that  may  pave  the  way  for  the 
commercial  development  of  reusable  launch  vehicles  by  reducing  cost  and 
increasing  space  transportation  reliability,  it  has  a  direct  relationship  with  the  key 
outcome  of  expanding  the  commercial  development  of  space.  Related  targets  and 
indicators  must  be  identified  to  enable  such  monitoring. 

•  The  cause  for  unmet  targets  can  often  be  traced  to  NASA’s  indicators  in  its  fiscal 
year  1999  performance  plan.  For  example,  over  half  of  the  performance  targets 
related  to  the  key  outcome  of  expanding  scientific  knowledge  of  the  Earth  system 
were  not  met  because  achievement  was  tied  to  the  fiscal  year  1999  launch  of  the 
Terra  spacecraft  and  to  the  successful  operation  of  the  third  stratospheric  aerosol 
and  gas  experiment.  However,  although  a  significant  number  of  targets  were  not 
met  in  fiscal  year  1999,  most  are  recoverable  in  fiscal  year  2000.  The  Terra 
spacecraft  was  launched  in  December  1999,  and  NASA  anticipates  that  data 
collections  associated  with  the  fiscal  year  1999  targets  related  to  this  key  outcome 
will  be  accomplished  in  fiscal  year  2000. 

NASA’s  fiscal  year  1999  performance  report  and  fiscal  year  2001  performance  plan 
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describe  goals  and  measures  for  8  of  NASA’s  10  management  challenges.  However, 
both  the  plan  and  report  fail  to  provide  enough  specific  information  on  some  of  these 
management  challenges  to  fully  assess  NASA’s  actions.  We  have  the  following 
additional  observations  on  NASA’s  treatment  of  its  management  challenges  in  its 
fiscal  year  1999  performance  report  and  fiscal  year  2001  performance  plan: 

•  The  performance  report  does  not  provide  enough  specific  information  on  four  of 
these  management  challenges  to  fully  assess  the  extent  to  which  NASA  has 
addressed  them.  For  example,  NASA’s  performance  report  provides  no  specific 
information  regarding  NASA’s  progress  in  overcoming  the  previously  reported 
problems  with  fielding  the  Earth  Observing  System  Data  and  Information  System 
(EOSDIS) . 

•  The  performance  plan  also  does  not  provide  enough  specific  information  on  four 
of  these  management  challenges  to  fully  assess  the  extent  to  which  NASA  has 
addressed  them.  For  example,  while  NASA’s  performance  plan  has  a  target  to 
enhance  information  technology  security,  we  cannot  fully  assess  NASA’s  actions 
because  the  plan  lacks  sufficient  information. 

•  NASA  did  not  directly  address  2  of  the  10  management  challenges,  namely  (1) 
International  Space  Station  program  cost  control  and  (2)  closer  cooperation 
between  NASA  and  the  Department  of  Defense  and  the  development  of  a  national 
perspective  on  aerospace  test  facilities. 

•  NASA  made  a  major  change  in  its  approach  to  the  contract  management  challenge 
after  the  reports  were  issued.  Because  of  difficulties  encountered  in  developing 
an  integrated  financial  management  system — a  critical  component  to  enhancing 
contract  management — NASA  has  delayed  agencywide  deployment  of  the  system. 
As  a  result,  targets  established  for  fiscal  year  2001  cannot  be  achieved. 

Objectives,  Scope,  and  Methodology 

Our  objectives  concerning  selected  key  agency  outcomes  were  to  (1)  identify  and 
assess  the  quality  of  the  performance  goals  and  measures  directly  related  to  a  key 
outcome,  (2)  assess  the  agency’s  actual  performance  in  fiscal  year  1999  for  each 
outcome,  and  (3)  assess  the  agency’s  planned  performance  for  fiscal  year  2001  for 
each  outcome.  Our  objectives  concerning  major  management  challenges  were  to  (1) 
assess  how  well  the  agency’s  fiscal  year  1999  performance  report  discussed  the 
progress  it  had  made  in  resolving  the  major  management  challenges  that  we  and  the 
agency’s  Inspector  General  had  previously  identified  and  (2)  identify  whether  the 
agency’s  fiscal  year  2001  performance  plan’s  goals  and  measures  were  applicable  to 
the  major  management  challenges.  As  agreed,  to  meet  the  Committee’s  tight 
reporting  time  frames,  we  generally  based  our  observations  on  the  requirements  of 
GPRA,  guidance  to  agencies  from  the  Office  of  Management  and  Budget  for 
developing  performance  plans  and  reports  (OMB  Circular  A-ll,  Part  2),  previous 
reports  and  evaluations  by  us  and  others,  our  knowledge  of  NASA’s  operations  and 
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programs,  and  our  observations  on  NASA’s  other  GPRA-related  efforts.  We  did  not 
independently  verify  the  information  contained  in  the  performance  report  or  plan. 

We  conducted  our  review  from  April  through  May  2000  in  accordance  with  generally 
accepted  government  auditing  standards. 

Agency  Comments  and  Our  Evaluation 

In  written  comments  to  a  draft  of  this  report,  NASA’s  Associate  Deputy  Administrator 
commented  on  two  issues:  (1)  the  usefulness  of  the  revised  approach  in  the  Fiscal 
Year  2001  Performance  Plan  for  reporting  on  progress  and  (2)  data  validation  and 
verification. 

Concerning  reporting  on  progress,  NASA  expressed  concern  that  we  had  not  taken 
the  opportunity  to  assess  the  agency’s  performance  report  and  plans  in  the  context  of 
the  goals  and  missions  it  is  striving  to  achieve.  NASA  stated  that  it  tried  to  strike  a 
balance  between  current  expenditures  and  the  achievement  of  longer-term  goals  and 
objectives  and  to  put  successes  and  failures  in  an  appropriate  context.  Because 
outcomes  in  research  and  development  are  not  realized  for  lengthy  periods,  NASA 
felt  that  it  was  important  to  communicate  annual  progress  towards  achievement  of 
those  goals  to  make  clear  the  relationship  between  the  current  investment  and  the 
achievement  of  longer-term  goals.  NASA  also  commented  that  the  fiscal  year  2001 
plan  aggregates  a  series  of  individual  programmatic  outputs  to  demonstrate  trends  in 
activities  leading  to  longer-term  goals. 

We  believe  that  we  have  appropriately  assessed  NASA’s  performance  report  and 
plans  in  the  context  of  the  goals  and  missions  the  agency  is  trying  to  achieve.  Also, 
we  did  not  conclude,  as  NASA  indicated  in  its  comments,  that  NASA’s  revised 
approach  to  the  Fiscal  Year  2001  Performance  plan  will  not  provide  improved 
reporting  of  progress.  Our  specific  concern  is  that  the  revised  approach  allows  NASA 
to  assess  a  performance  target  as  having  been  met  even  if  some  of  the  performance 
indicators  supporting  the  target  have  not  been  achieved.  We  maintain  that  this 
approach  will  require  NASA  to  provide  convincing  evidence  that  a  desired  outcome  is 
met  despite  the  nonachievement  of  one  or  more  of  the  supporting  indicators. 
Regarding  NASA’s  views  on  aggregate  programmatic  outputs,  we  recognize  that  some 
output  measures  are  necessary  to  show  progress  or  contribution  to  intended  results. 
However,  continued  reliance  primarily  on  output  measures  places  added  burden  on 
NASA  to  continuously  demonstrate  the  linkages  between  program  efforts  and  results 
and  make  improvements  to  strengthen  such  linkages.  Moving  toward  outcome- 
oriented  measures  for  important  goals  and  objectives  will  reduce  this  burden. 

Regarding  the  credibility  of  its  data,  NASA  stated  that  it  relies  on  individuals 
responsible  for  the  performance  to  validate  and  verify  the  information  provided  for 
GPRA  compliance  and  on  the  NASA  Advisory  Council  to  independently  evaluate 
progress.  NASA  believes  we  did  not  acknowledge  this  independent  evaluation  by 
subject  matter  experts,  although  we  did  discuss  the  Council’s  remarks  in  subsequent 
sections.  Our  primary  point  is  that  NASA’s  performance  report  does  not  provide  the 
reader  a  perspective  of  actions  NASA  is  taking  to  ensure  that  the  information  is 


Page  5 


GAO/NSIAD-00-192R  NASA’s  FY  1999  Performance  and  FY  2001  Performance  Plan 


B-285486 


accurate  and  that  data  limitation  issues  and  problems  have  been  adequately 
addressed.  Communicating  verification  and  validation  approaches  provides  greater 
confidence  that  the  reported  performance  information  is  credible  and  enhances  the 
usefulness  of  the  information.  At  the  same  time,  making  congressional  and  other 
decisionmakers  aware  of  significant  data  limitations  and  their  implications  allows 
them  to  properly  judge  the  significance  of  results  achieved.  We  have  revised  the 
report  to  reflect  the  Advisory  Council’s  independent  evaluation  role. 

NASA  also  provided  suggestions  to  improve  the  technical  accuracy  of  the  report.  We 
incorporated  those  suggestions  where  appropriate. 

NASA’s  written  comments  are  presented  in  enclosure  III. 


As  arranged  with  your  office,  unless  you  publicly  announce  the  contents  earlier,  we 
plan  no  further  distribution  of  this  report  until  30  days  after  its  issue  date.  At  that 
time  copies  to  others  will  be  available  on  the  Internet  at  http://www.gao.gov. 

Please  call  me  on  (202)  512-4841  if  you  or  your  staff  have  any  questions.  Key 
contributors  to  this  report  were  Richard  J.  Herley,  Shirley  B.  Johnson,  Charles  W. 
Malphurs,  John  A.  Deferrari,  and  Elizabeth  L.  Johnston. 


Allen  Li 

Associate  Director 
Defense  Acquisitions  Issues 
lia.nsiad@gao.gov 
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Observations  on  the  National  Aeronautics  and  Space  Administration’s  Fiscal 
Year  1999  Actual  Performance  and  Fiscal  Year  2001  Planned  Performance 
Related  to  Key  Outcomes 

This  enclosure  contains  our  observations  on  the  National  Aeronautics  and  Space 
Administration’s  (NASA)  fiscal  year  1999  actual  performance  and  fiscal  year  2001 
planned  performance  for  key  outcomes  identified  by  the  Senate  Governmental  Affairs 
Committee  as  important  mission  areas  for  the  agency.  The  key  outcomes  for  NASA 
are  to  (1)  expand  scientific  knowledge  of  the  Earth  system;  (2)  deploy  and  operate 
the  International  Space  Station  safely  and  cost-effectively;  and  (3)  expand  the 
commercial  development  of  space.  As  requested  we  have  identified  the  goals  and 
measures  directly  related  to  a  selected  key  outcome.  Our  observations  are  organized 
according  to  each  selected  key  outcome  and  follow  the  goals  and  measures. 

Key  Agency  Outcome:  Expand  Scientific  Knowledge  of  the  Earth  System 

Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  Scientific  Knowledge  of  the  Earth  System 

Objective:  Understand  the  causes  and  consequences  of  land-cover/land-use  change. 
Targets: 

(1)  Begin  to  refresh  the  global  archive  of  30-meter  land  imagery  from  Landsat  7,  two 
to  three  times  per  year.  A  single  global  archive  has  not  been  constructed  since  the 
late  1970s.  Landsat  7  also  includes  a  15-meter  panchromatic  band  for  the  study  of 
ecosystems  disturbance. 

(Target  met.) 

(2)  Begin  to  collect  near-daily  global  measurements  of  the  terrestrial  biosphere  (an 
index  of  terrestrial  photosynthetic  processes  from  which  calculations  of  carbon 
uptake  are  made)  from  instruments  on  the  Earth  Observing  System  (EOS)  Terra  AM-1 
spacecraft. 

(Target  not  met.) 

(3)  Collect  near-daily  global  measurements  of  ocean  color  (an  index  of  ocean 
productivity  from  which  calculations  of  ocean  update  of  carbon  are  made) . 

(Target  not  met.) 

Objective:  Predict  seasonal-to-interannual  climate  variations. 
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Targets: 

(4)  Begin  the  second  of  a  3-year  sequence  of  instantaneous  measurements  of  rainfall 
rates  and  monthly  accumulations  in  the  global  tropics.  This  will  be  the  first-ever 
measurement  of  global  tropical  rainfall.  Current  uncertainty  in  global  tropical  rainfall 
estimates  is  50  percent.  Tropical  Rainfall  Measuring  Mission  (TRMM)  data  will 
reduce  this  uncertainty  to  10  percent,  an  80-percent  improvement. 

(Target  met.) 

(5)  Begin  the  measurement  of  sea-surface  wind  speed  and  direction  at  a  spatial 
resolution  of  25-kilometer  resolution  over  at  least  90  percent  of  the  ice-free  global 
oceans  every  2  days.  This  represents  a  resolution  increase  of  a  factor  of  two  and  a 
15-percent  increase  in  coverage  over  previous  measurements.  Data  from  this  mission 
will  be  used  to  improve  the  short-term  weather  forecasts. 

(Target  met.) 

Objective:  Identify  natural  hazards,  processes,  and  mitigation  strategies  for  floods, 
droughts,  and  volcanoes. 

(6)  Provide  instruments  sufficient  to  create  the  first  digital  topographic  map  of  80 
percent  of  the  Earth’s  land  surface  between  60  degrees  north  and  56  degrees  south. 
The  Shuttle  Radar  Topography  Mission  will  be  ready  for  launch  by  the  end  of  fiscal 
1999. 

(Target  met.) 

(7)  Use  the  Global  Positioning  System  (GPS)  array  in  southern  California  to  monitor 
crustal  deformation  on  a  daily  basis  with  centimeter  precision  and  initiate  the 
installation  of  the  next  100  stations.  The  data  will  be  archived  at  the  Jet  Propulsion 
Laboratory  (JPL)  and  run  in  models,  with  the  results  given  to  the  California  Seismic 
Safety  Commission  and  the  Federal  Emergency  Management  Agency  to  be  used  for 
earthquake  warning. 

(Target  met.) 

(8)  Ensure  that  data  received  from  GPS  receivers  in  low-earth  orbit  will  also  be  used 
to  test  improved  algorithms  for  measuring  atmosphere  temperature.  The  data  will 
serve  as  the  future  prototype  for  improving  short-term  weather  forecasts  globally. 

The  data  will  be  archived  at  JPL,  and  the  results  will  be  published  in  science 
literature. 

(Target  not  met.) 

Objective:  Detect  long-term  climate  change,  causes,  and  impacts. 

Targets: 

(9)  Begin  to  conduct  daily  observations  of  cloud  properties,  such  as  extent,  height, 
optical  thickness,  and  particle  size. 

(Target  not  met.) 


Page  8 


GAO/NSIAD-00-192R  NASA’s  FY  1999  Performance  and  FY  2001  Performance  Plan 


Enclosure  I 


(10)  Map  aerosol  formation,  distribution,  and  sinks  over  the  land  and  oceans. 

(Target  not  met.) 

(11)  Achieve  significant  reduction  in  the  uncertainty  in  components  of  the  earth’s 
radiation  balance  (that  is,  improved  angular  models  leading  to  an  estimated  error 
reduction  in  regional-scale,  monthly-  average  net  radiation  of  about  50  percent). 

(Target  not  met.) 

Objective:  Understand  the  causes  of  variation  in  ozone  concentrations  and 
distribution  in  the  upper  and  lower  atmosphere. 

Targets: 

(12)  Use  new  retrieval  methods  to  collect  and  analyze  three  new  data  products, 
including  surface  ultraviolet  radiation,  tropospheric  aerosols,  and  in  certain  regions 
tropospheric  columns.  Together  with  Solar  Backscatter  Ultraviolet  data,  there  will 
now  be  a  continuous  20-year  data  set  for  total  ozone  that  will  measure  the  ultimate 
effectiveness  of  the  Montreal  Protocol  on  substances  that  deplete  the  ozone  layer. 
These  data  are  also  useful  in  routing  aircraft  around  areas  of  concentrated  volcanic 
dust.  These  new  and  extended  data  products  will  be  made  available  on  the  Total 
Ozone  Mapping  Spectrometer  Web  site  for  dissemination  and  access  to  a  broader 
community  than  just  NASA-sponsored  scientists. 

(Target  met.) 

(13)  Improve  the  collection  and  analysis  of  measurements  provided  by  the 
Stratospheric  Aerosol  and  Gas  Experiment  (SAGE  II) .  These  improvements  include 
lunar  occultation  capability,  allowing  for  new  nitrogen  trioxide  and  chlorine  dioxide 
measurements;  additional  wavelength  sampling,  providing  direct  measurements  and 
ability  to  retrieve  aerosols  throughout  the  troposphere;  and  appreciably  higher 
spectral  resolution,  allowing  significantly  improved  distributions  of  water  vapor  and 
ozone  in  the  upper  troposphere  and  lower  stratosphere.  This  represents 
approximately  a  two-thirds  reduction  in  error  in  near-tropopause  water  vapor 
measurements  as  well  as  extension  of  ozone  measurements  into  the  midtroposphere 
with  10-  to  15-percent  errors.  Such  data  were  not  available  before. 

(Target  not  met.) 

(14)  Initiate  the  full  Southern  Hemisphere  Additional  Ozonesonde  network  to  obtain 
the  first-ever  climatology  of  the  upper  troposphere  ozone  in  the  tropics. 

(Target  met.) 

(15)  Continue  the  detailed  multi-aircraft  study  of  troposphere  chemistry  over  the 
tropical  Pacific  Ocean,  especially  the  contribution  of  long-range  transport  of  air  from 
South  America  and  Africa  to  otherwise  unpolluted  areas.  Complete  the  field 
measurements  phase  of  the  Pacific  Exploratory  Mission  (PEM) -Tropics  B  (rainy 
season)  with  an  improved  payload  that  has  resulted  from  an  initiative  to  develop  a 
smaller,  lighter  payload  with  equal  or  better  performance  than  PEM-Tropics-A  (dry 
season).  The  results  will  be  fully  analyzed  and  published. 

(Target  not  met.) 
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(16)  Measure  surface  levels  of  chlorine-  and  bromine-containing  chemical  compounds 
addressed  under  the  Montreal  Protocol  to  document  the  decreasing  concentrations  of 
the  regulated  compounds  and  the  rising  concentrations  of  their  replacements  to 
quantify  the  decrease  in  total  halogen  abundance  in  the  lower  atmosphere.  The  data 
will  be  provided  to  researchers  supporting  the  World  Meteorological  Organization 
assessment  process. 

(Target  met.) 

Objective:  Successfully  launch  spacecraft. 

(17)  Successfully  launch  three  spacecraft,  within  10  percent  of  budget  on  average. 

(Target  not  met.) 

GAO  Observations  on  NASA’s  Fiscal  Year  1999  Actual  Performance  for  the  Key 
Agency  Outcome  to  Expand  Scientific  Knowledge  of  the  Earth  System 

The  associated  performance  objectives  and  targets  adequately  indicate  overall 
progress  toward  the  related  outcome  and  are  generally  objective  and  measurable. 
However,  for  the  most  part,  the  performance  targets  appear  to  be  output  measures. 
NASA  notes  in  its  fiscal  year  1999  performance  report  that  it  has  difficulty  in 
quantifying  outcomes  and  relating  current  outcomes  to  current  fiscal  expenditures 
because  of  the  long-term  character  of  its  objectives.  NASA’s  use  of  output  measures 
places  the  burden  on  NASA  to  ensure  that  it  continuously  links  program  targets  and 
projected  outcomes  and  makes  the  improvements  needed  to  strengthen  such 
linkages.  The  performance  information  clearly  articulates  the  degree  to  which  the 
annual  targets  have  been  achieved  and  are  directly  relevant  to  the  associated 
performance  targets.  However,  some  of  the  target  descriptions  should  be  more 
concise.  While  we  recognize  that  scientific  phenomena  are  highly  technical  in  nature, 
greater  attention  should  be  paid  to  explaining  targets  and  outcomes  in  “plain 
English.”  The  performance  report  includes  all  performance  targets  established  in  the 
fiscal  year  1999  performance  plan  for  this  outcome;  no  targets  were  excluded  or 
revised.  NASA  met  slightly  less  than  half  of  its  targets  for  this  key  outcome.  Only 
one  objective,  “Predict  seasonal-to-interannual  climate  variations,”  had  been  fully 
met.  For  one  objective,  none  of  its  targets  were  met.  The  other  objectives  had  a  mix 
of  met  and  unmet  targets. 

The  performance  report  includes  an  independent  evaluation  completed  in  February 
2000  by  the  NASA  Advisory  Council.  According  to  the  performance  report,  the 
Council  provided  a  qualitative  evaluation  of  NASA’s  fiscal  year  1999  performance, 
going  beyond  the  question  of  whether  NASA  met  specific  performance  targets  in  the 
fiscal  year  1999  performance  plan.  The  Council  indicated  that  the  objectives  related 
to  this  outcome  were  achieved  and  highlighted  performance  targets  1  and  5  as  key 
targets  met  during  fiscal  year  1999. 

Discussion  of  data  credibility  is  limited  to  stating  that  each  of  the  program  and 
project  managers  are  fully  accountable  for  the  accuracy  of  the  performance 
information.  NASA’s  statement  does  not  provide  reasonable  assurance  that  the 
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performance  information  is  credible.  Our  observation,  applicable  to  all  key  outcomes 
we  reviewed,  is  that  the  report  would  be  more  useful  to  decisionmakers  if  NASA,  at  a 
minimum,  described  procedures  used  to  verify  and  validate  performance  information 
and  addressed  data  limitation  problems  and  issues  in  its  performance  data. 

Discussion  of  the  verification  and  validation  of  supporting  data,  as  well  as  limitations 
to  that  data,  would  be  useful  in  fully  understanding  the  agency’s  progress.  Over  the 
past  several  years,  we  have  identified  NASA’s  failure  to  include  explicit  procedures  it 
would  use  to  verify  and  validate  performance  and  address  possible  data  limitations  in 
its  annual  performance  plans.  A  recent  NASA  Office  of  Inspector  General  (OIG) 
report  also  found  weaknesses  in  the  accuracy  and  reliability  of  fiscal  year  1999 
performance  targets  (145)  in  NASA’s  performance  report.2  The  OIG  found  that  5  (22 
percent)  of  the  23  performance  targets  (unrelated  to  this  outcome)  it  reviewed  were 
not  fully  reliable  because  the  supporting  data  did  not  accurately  support  the  results 
described.  From  the  audit  results,  the  OIG  concluded  that  other  targets  may  also 
have  inaccurate  supporting  data  and  reported  results.  A  NASA  official  said  that 
NASA  had  corrected  or  clarified  specific  performance  results  in  the  fiscal  year  1999 
performance  report  to  address  the  OIG’s  concerns. 

Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency 
Outcome  to  Expand  Scientific  Knowledge  of  the  Earth  System 

(2)  Begin  to  collect  near-daily  global  measurements  of  the  terrestrial  biosphere  (an 
index  of  terrestrial  photosynthetic  processes  from  which  calculations  of  carbon 
uptake  are  made)  from  instruments  on  the  Earth  Observing  System  Terra  AM-1 
spacecraft. 

(3)  Collect  near-daily  global  measurements  of  ocean  color  (an  index  of  ocean 
productivity  from  which  calculations  of  ocean  update  of  carbon  are  made) . 

(8)  Ensure  that  data  received  from  GPS  receivers  in  low-earth  orbit  will  also  be  used 
to  test  improved  algorithms  for  measuring  atmosphere  temperature. 

(9)  Begin  to  conduct  daily  observations  of  cloud  properties  such  as  extent,  height, 
optical  thickness,  and  particle  size. 

(10)  Map  aerosol  formation,  distribution,  and  sinks  over  the  land  and  oceans. 

(11)  Achieve  significant  reduction  in  the  uncertainty  in  components  of  the  earth’s 
radiation  balance. 

(13)  Improve  the  collection  and  analysis  of  measurements  provided  by  the  SAGE  II. 

(15)  Continue  the  detailed  multi-aircraft  study  of  troposphere  chemistry  over  the 
tropical  Pacific  Ocean,  especially  the  contribution  of  long-range  transport  of  air  from 


2  Validating  FY  1999  Performance  Data  To  Be  Reported  Under  the  Government  Performance  Results 
Act  (GPRA) ,  (IG-00-020,  Mar.  28,  2000). 
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South  America  and  Africa  to  otherwise  unpolluted  areas.  Complete  the  field 
measurements  phase  of  the  PEM-Tropics  B  (rainy  season)  with  an  improved  payload 
that  has  resulted  from  an  initiative  to  develop  a  smaller,  lighter  payload  with  equal  or 
better  performance  than  PEM-Tropics-A  (dry  season) . 

(17)  Successfully  launch  three  spacecraft,  within  10  percent  of  the  budget  on  average. 

GAO  Observations  on  NASA’s  Unmet  Fiscal  Year  1999  Performance  Goals  and 
Measures  for  the  Key  Agency  Outcome  to  Expand  Scientific  Knowledge  of  the  Earth 
System 

The  performance  assessments  indicate  that  targets  8  and  1 5  were  only  partially  met. 
The  assessments  provide  a  reasonable  explanation  as  to  why  target  15  was  not  fully 
met  but  does  not  provide  a  clear  explanation  of  the  reason  target  8  was  not  fully  met. 
As  a  result,  the  impact  of  not  having  data  from  Global  Positioning  System  receivers  to 
test  improved  algorithms  for  measuring  atmosphere  temperature  on  this  key  outcome 
cannot  be  determined,  and  corrective  actions  cannot  be  assessed  at  this  time.  The 
assessment  describes  clear  and  reasonable  actions  that  will  be  taken  to  complete 
target  15.  While  a  reasonable  time  frame  is  provided  for  completion  of  target  15,  no 
time  frame  is  indicated  for  completion  of  target  8. 

The  performance  assessments  indicate  that  targets  2,  3,  9, 10, 11, 13,  and  17  were  not 
met.  NASA  provided  reasonable  explanations  for  not  achieving  these  targets  and 
reasonable  time  frames  for  achieving  them.  The  nonachievement  of  these  targets  was 
tied  to  the  delay  associated  with  the  Terra  launch  or  issues  related  to  SAGE  III. 

Target  17  referred  to  three  launches;  one  launch  was  Terra,  which  was  delayed. 
Achievement  of  that  launch  along  with  the  other  six  targets  was  deferred  to  fiscal 
year  2000.  Target  17  was  written  for  Terra  to  deliver  specific  performance  levels  for 
fiscal  year  1999 — launch  within  10  percent  of  the  budget.  NASA  had  no  opportunity 
to  launch  Terra  within  10  percent  of  the  budget  in  fiscal  year  1999  because  the  launch 
was  delayed.  Therefore,  target  17  was  determined  to  be  unachievable  because 
opportunity  to  perform  at  the  specified  levels  had  passed.  Terra  was  later  launched, 
in  December  1999,  and  NASA  anticipates  that  data  collections  associated  with  the 
fiscal  year  1999  targets  related  to  this  key  outcome  will  be  accomplished  in  fiscal  year 
2000. 

Fiscal  Year  2000  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  Scientific  Knowledge  of  the  Earth  System 

In  our  view,  21  of  the  27  fiscal  year  2000  performance  targets  are  new  for  this 
outcome.  They  are  as  follows: 

■  Sensor  Intercomparison  and  Merger  for  Biological  and  Interdisciplinary  Ocean 
Studies  will  merge  Moderate  Resolution  Imaging  Spectrometer  (MODIS)  ocean 
color  data  into  the  global  ocean  color  time  series,  which  began  with  the  Ocean 
Color  Temperature  Sensor  and  the  Sea-viewing  Wide  Field-of-view  Sensor.  Use 
the  time  series  to  understand  and  predict  the  response  of  the  marine  ecosystem  to 
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climate  change.  The  data  set  will  be  made  available  via  the  Goddard  Space  Flight 
Center  Distributed  Active  Archive  Center. 

■  Produce  near-real-time  fire  monitoring  and  impact  assessments  based  on  Landsat 
and  EOS  inventory  and  process  monitoring.  These  assessments  will  provide  an 
observational  foundation  for  monitoring  changes  in  ecosystem  productivity  and 
disturbance.  Near-real-time  assessments  will  be  posted  on  a  Web  site  for  quick 
access  by  researchers  and  regional  authorities. 

■  Establish  a  benchmark  for  global  and  regional  rainfall  measurements  by 
combining  TRMM  measurements  with  measurements  from  other  sources.  Maps 
of  the  diurnal  cycle  of  precipitation  will  be  created  for  the  first  time.  The  existing 
10-year  data  set  will  be  combined  with  TRMM  measurements  to  validate  climate 
models  and  demonstrate  the  impact  of  rainfall  on  short-term  weather  forecasting. 
The  data  will  be  distributed  through  the  Goddard  Space  Flight  Center  Distributed 
Active  Archive  Center  for  ease  of  access  to  science  and  operational  users. 

■  Develop  and  improve  methods  to  couple  state-of-the-art  land  surface  and  sea  ice 
models  to  a  global,  coupled  ocean-atmosphere  model  and  use  them  to  predict 
regional  climatic  consequences  of  El  Nino  or  La  Nina  in  the  tropical  Pacific.  The 
results  of  this  research  will  be  published  in  open  literature  and  provided  to  the 
National  Oceanic  and  Atmospheric  Administration’s  National  Climate  Prediction 
Center  and  the  U.S.  Navy’s  Fleet  Numeric  Prediction  Center.  The  ultimate  goal  is 
to  develop  a  capability  to  significantly  improve  the  prediction  of  seasonal-to- 
interannual  climate  variations  and  their  regional  climate  consequences.  The  main 
focus  is  on  North  America. 

■  Measure  production  and  radiactive  properties  of  aerosols  produced  by  biomass 
burning  in  Africa  based  on  SAFARI  2000  (field  experiment)  and  EOS  instruments. 
This  will  include  extensive  international  participation.  This  burning  is  estimated 
to  contribute  one  half  of  all  global  atmospheric  aerosols. 

*  Launch  the  NASA-Centre  Nationale  d’Etudes  Spatiales  (CNES)  Jason- 1  mission. 
This  follow-on  to  the  Ocean  Topography  Experiment/Poseidon  is  to  achieve  a 
factor-of-4  improvement  in  accuracy  in  measuring  ocean  basin-scale,  sea-level 
variability.  This  is  one  order  of  magnitude  better  than  that  specified  for  the  Ocean 
Topography  Experiment/Poseidon. 

■  Generate  the  first  basin-scale,  high-resolution  estimate  of  the  state  of  the  Pacific 
Ocean  as  part  of  the  international  Global  Ocean  Data  Assimilation  Experiment. 

■  Use  southern  California  GPS  array  data  to  understand  the  connection  between 
seismic  risk  and  crustal  strain  leading  to  earthquakes. 

■  Develop  models  to  use  time-varying  gravity  observations  for  the  first  time  in 
space. 
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•  Demonstrate  the  utility  of  space-borne  data  for  flood-plain  mapping  with  the 
Federal  Emergency  Management  Agency. 

■  Develop  an  automatic  volcano  cloud/ash  detection  algorithm  employing  EOS  data 
sets  for  use  by  the  Federal  Aviation  Administration. 

«  Complete  the  collection  of  satellite  data  needed  for  the  17-year  cloud  climatology 
being  developed  under  the  International  Satellite  Cloud  Climatology  Project. 

These  data  will  be  used  to  improve  the  understanding  and  modeling  of  the  role  of 
clouds  in  climate  and  will  be  available  through  the  Goddard  Distributed  Active 
Archive  Center. 

■  Provide  for  the  continuation  of  the  long-term,  precise  measurement  of  the  total 
solar  irradiance  with  the  launch  of  the  EOS  Active  Cavity  Radiometer  Irradiance 
Monitor. 

■  Acquire,  through  a  Radarsat  repeat  of  the  Antarctic  Mapping  Mission  conducted  in 
September-October  1997,  a  second  set  of  high-resolution  radar  data  over  all  of 
Antarctica  for  comparison  with  the  baseline  data  set  acquired  in  1997,  to  identify 
changes  on  the  ice  sheet. 

■  Publish  the  first  detailed  estimates  of  thickening/thinning  rates  for  all  major  ice 
drainage  basins  of  the  Greenland  ice  sheet,  derived  from  repeated  airborne  laser- 
altimetry  surveys.  These  measures  will  represent  the  baseline  data  set  to  compare 
with  early  Geoscience  Laser  Altimeter  System  data  (July  2001  launch). 

■  Initiate  a  program  of  airborne  mapping  of  layers  within  the  Greenland  ice  sheet  to 
decipher  the  impact  of  past  climate  variations  on  polar  regions. 

■  Develop  a  remote-sensing  instrument/technique  for  ocean  surface  salinity 
measurements  from  aircraft.  The  goal  is  to  improve  measurement  accuracy  to 
one  order  of  magnitude  better  than  available  in  fiscal  year  1998.  The  ultimate  goal 
is  the  capability  to  measure  sea  surface  salinity  globally  from  space. 

■  Continue  to  improve  the  design  and  sophistication  of  a  global  climate  system 
model,  including  the  use  of  higher  resolution,  to  make  it  a  state-of-the-art  climate 
system  model  for  projecting  the  climatic  consequences  at  the  regional  level. 
Evidence  of  improvement  will  be  increased  resolution  from  added  computing 
power  and  better  numerical  representations. 

■  Implement  the  SAGE  111  Ozone  Loss  and  Validation  Experiment.  Measurements 
will  be  made  from  October  1999  to  March  2000  in  the  Arctic/high-latitude  region 
from  the  NASA  DC-8  and  ER-2  aircraft  and  balloon  platforms.  These  tools  will 
help  acquire  correlative  data  to  validate  SAGE  111  data  and  assess  high-latitude 
ozone  loss. 

■  Complete  the  analysis  and  publication  of  the  PEM-Tropics-B  field  experiment. 
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■  Complete  the  troposphere  chemistry  aircraft  instrument  size  and  weight 
reductions  (by  approximately  40  percent)  initiative. 

In  our  view,  at  least  6  of  the  27  targets  are  variations  of  fiscal  year  1999  targets.  They 
are  as  follows: 

■  Continue  the  development  of  a  global  land-cover/use  change  data  set  based  on 
Landsat  and  EOS  instruments,  at  a  seasonal  refresh  rate  (variation  of  fiscal  year 
1999  target  1). 

■  Continue  to  collect  near-daily  global  measurements  of  the  terrestrial  biosphere 
(an  index  of  terrestrial  photosynthetic  processes  from  which  calculations  of 
carbon  uptake  are  made)  from  instruments  on  the  EOS  AM-1  spacecraft  (variation 
of  fiscal  year  1999  target  2). 

■  Continue  the  ocean  color  time  series  with  60-percent  global  coverage  every  4 
days — a  35-percent  improvement  over  FY  1999  (variation  of  fiscal  year  1999  target 
3). 

■  Continue  the  development  of  the  global  aerosol  climatology  data  set  and  analysis 
of  this  climatology  in  climate  models.  The  data  will  be  available  through  the 
Goddard  Space  Flight  Center  Distributed  Active  Archive  Center  (variation  of 
fiscal  year  1999  target  10). 

■  Complete  the  planning  for  major  new  2001  airborne/unmanned  aerospace  vehicle 
missions  that  will  use  a  smaller  troposphere  chemistry  aircraft  instrument 
(variation  of  fiscal  year  1999  target  15). 

■  Successfully  launch  three  spacecraft  and  deliver  two  instruments  for  international 
launches  within  10  percent  of  budget  on  average  (variation  of  fiscal  year  1999 
target  17). 

GAO  Observations  on  NASA’s  Fiscal  Year  2000  Planned  Performance  for  the  Key 
Agency  Outcome  to  Expand  Scientific  Knowledge  of  the  Earth  System 

The  performance  report  does  not  provide  an  assessment  of  the  effect  of  fiscal  year 
1999  performance  on  estimated  performance  levels  for  fiscal  year  2000.  For  example, 
it  does  not  discuss  (1)  what  goals  or  measures  will  be  revised  to  be  more  outcome 
oriented,  (2)  whether  NASA  will  revise  the  means  and  strategies  section  of  its  fiscal 
year  2000  performance  plan  to  better  show  the  achievement  of  specific  performance 
goals,  or  (3)  whether  NASA  will  make  changes  to  its  verification  and  validation 
practices  and  procedures  in  the  fiscal  year  2000  performance  plan  to  improve  the 
credibility  of  the  agency’s  performance  information. 

Of  the  27  performance  targets  for  fiscal  year  2000,  21  are  new.  In  previous  years, 
NASA  officials  have  stated  that  performance  targets  may  change  from  one  fiscal  year 
to  another  because  some  are  discrete  events  that  take  place  within  a  fiscal  year,  some 
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change  with  the  phase  of  a  program,  and  others  are  expected  to  be  achieved  in  a 
specific  fiscal  year  with  no  carryover  to  a  subsequent  fiscal  year.  Therefore,  many  of 
NASA’s  performance  targets  are  new  each  fiscal  year.  We  believe  this  is  reasonable  if 
NASA  establishes  performance  measures  that  cover  significant  efforts  and  critical 
management  issues  and  problems.  However,  including  the  rationale  for  newly 
established  targets  in  annual  performance  plans  and  reports  would  clarify  the  reasons 
for  the  new  targets. 

Fiscal  Year  2001  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  Scientific  Knowledge  of  the  Earth  System 

In  our  view,  8  of  the  11  fiscal  year  2001  performance  targets  are  new  for  this 
outcome.  They  are  as  follows: 

■  Explore  the  dynamics  of  the  global  carbon  cycle  by  developing,  analyzing,  and 
documenting  at  least  three  multiyear  data  set.  An  example  of  this  will  be 
development  of  a  global  time  series  of  phytoplankton  biomass  and  primary 
productivity  in  the  oceans. 

■  Explain  the  dynamics  of  the  global  carbon  cycle  by  building  improved  models  and 
prediction  capabilities.  One  indicator  of  this  activity  is  improving  ecological 
models  needed  to  predict  ecosystem  responses  to  global  environmental  changes 
by  15  percent. 

■  Explore  the  dynamics  of  global  water  cycle  by  developing,  analyzing,  and 
documenting  at  least  one  multiyear  data  set  such  as  that  needed  to  obtain 
accurate  maps  of  the  sunrise  to  sunset  changes  in  precipitation. 

■  Explain  the  dynamics  of  the  global  water  cycle  by  building  improved  models  and 
prediction  capabilities,  specifically  improving  current  understanding  of  the  large- 
scale  effects  of  clouds  on  climate. 

■  Explain  the  dynamics  of  long-term  climate  variability  by  building  improved 
models  and  prediction  capabilities.  One  example  of  this  activity  will  be 
demonstrating  experimental  seasonal  climate  predictions  by  using  next 
generation  super  computing  systems  and  new-coupled  air-ocean-ice  models. 

■  Explore  the  dynamics  of  atmospheric  composition  by  developing,  analyzing,  and 
documenting  at  least  three  multiyear  data  sets,  such  as  providing  continuity  of 
multidecadal  total  ozone  concentration  measurements. 

■  Explain  the  dynamics  of  atmospheric  chemistry  by  building  improved  models  and 
prediction  capabilities.  One  example  of  this  activity  will  be  characterizing  the 
atmospheric  plume  flowing  out  of  East  Asia,  its  evolution  as  it  transits  eastward 
over  the  Pacific,  and  its  contribution  to  global  atmospheric  chemical  composition. 
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■  Explore  the  dynamics  of  the  earth’s  interior  crust  by  developing,  analyzing,  and 
documenting  at  least  one  multiyear  data  set,  such  as  analysis  of  30-meter 
topographic  data  for  global  geological  and  geomorphic  process  studies  and 
improved  mapping  of  terrain  features. 

In  our  view,  at  least  3  of  the  11  targets  are  variations  of  fiscal  year  1999  and/or  2000 
performance  targets.  They  are  as  follows: 

■  The  Earth  Science  Enterprise  will  successfully  develop,  have  ready  for  launch, 
and  operate  instruments  on  at  least  two  spacecraft  within  10  percent  of  their 
schedules  and  budget  to  enable  earth  science  research  and  applications  goals  and 
objectives  (variation  of  fiscal  year  1999  target  17  and  related  fiscal  year  2000 
target  “successfully  launch  spacecraft  and  deliver  2  instruments  for  international 
launches  within  10  percent  of  budget  on  average”). 

■  Explore  the  dynamics  of  long-term  climate  variability  by  developing,  analyzing, 
and  documenting  at  least  two  multiyear  data  sets.  An  example  is  continuing  the 
high-precision,  multidecadal  record  of  total  solar  irradiance,  providing  a 
quantitative  understanding  of  the  solar  forcing  effects  on  the  earth’s  climate 
(variation  of  fiscal  year  2000  target  “Provide  for  the  continuation  of  the  long-term, 
precise  measurement  of  the  total  solar  irradiance  with  the  launch  of  EOS 
ACRIM”). 

■  Explain  the  dynamics  of  the  earth’s  interior  and  crust  by  building  improved 
models  and  prediction  capabilities.  An  indicator  of  this  activity  will  be  providing 
a  basis  for  future  tectonic  modeling  and  earthquake  vulnerability  assessment 
through  completion  of  the  southern  California  integrated  GPS  network,  (variation 
of  fiscal  year  2000  target  “Use  southern  California  GPS  array  data  to  understand 
the  connection  between  seismic  risk  and  crustal  strain  leading  to  earthquakes”) 

GAO  Observations  on  NASA’s  Fiscal  Year  2001  Planned  Performance  for  the  Key 
Agency  Outcome  to  Expand  Scientific  Knowledge  of  the  Earth  System 

NASA  revised  its  approach  to  the  development  of  performance  targets  in  the  fiscal 
year  2001  performance  plan.  What  are  now  known  as  indicators  were  considered 
performance  targets  in  the  fiscal  years  1999  and  2000  performance  plans.  The  targets 
are  now  written  more  generally,  and  one  or  more  indicators  may  support  a  particular 
performance  target.  For  example,  the  fiscal  year  2001  target  to  “explore  the 
dynamics  of  long  term  climate  variability  by  developing,  analyzing,  and  documenting 
multi-year  data  sets”  has  three  performance  indicators.  One  of  the  three  indicators 
provides  for  the  use  of  Jason- 1  satellite  data  to  continue  the  measurement  of  ocean 
basin-scale,  sea-level  variability  and  reducing  errors  to  less  than  3  centimeters. 

The  indicators  in  the  fiscal  year  2001  performance  plan  are  further  defined  in  an 
appendix  to  the  plan.  The  performance  plan  states  that  (1)  the  targets  have  been 
developed  to  enable  a  better  understanding  of  how  the  specific  measures  of  output 
(indicators)  contribute  to  the  eventual  outcomes  that  are  the  result  of  a  number  of 
years  of  research,  development,  and  data  analysis  and  (2)  the  targets  enable  NASA  to 
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display  indicators  from  multiple  years,  which  will  contribute  to  the  achievement  of 
the  summary  targets.  This  is  a  significant  change.  A  NASA  official  said  that 
nonattainment  of  one  of  several  indicators  will  not  disqualify  NASA  from  claiming 
achievement— a  response  to  NASA’s  experience  with  the  fiscal  year  1999 
performance  report,  in  which  targets  had  to  be  characterized  as  unmet  when  any 
associated  measure  was  not  attained.  We  believe  that  this  approach  will  require 
NASA  to  provide  convincing  evidence  that  a  desired  outcome  was  met  despite  the 
nonachievement  of  one  or  more  of  the  supporting  indicators. 

Key  weaknesses  that  we  identified  in  NASA’s  fiscal  year  1999  and  2000  performance 
plans  related  to  this  outcome  were  that  the  plans  did  not  provide  a  clear  rationale  for 
how  information  technology  (IT) -related  strategies  and  programs  contribute  to 
achievement  of  performance  goals  and  did  not  discuss  procedures  for  verifying  and 
validating  performance  data. 

In  general,  NASA’s  fiscal  year  2001  performance  plan  still  does  not  provide  a  clear 
rationale  for  how  IT-related  strategies  and  programs  will  contribute  to  achievement 
of  NASA’s  goals  or  show  any  allocation  of  IT-related  dollars  and  personnel  to 
performance  goals.  Goals  for  managing  information  technology  across  the  agency 
are  stated  in  terms  of  broad  categories  for  improvement,  such  as  increased  capability 
and  efficiency  and  enhanced  security,  but  do  not  include  any  quantitative  measures. 
The  one  exception  is  the  goal  of  increasing  dissemination  of  earth  science  data, 
which  is  accomplished  through  the  EOS  Data  and  Information  System.  This  goal  was 
not  included  in  our  assessment  of  this  key  outcome.  The  plan  sets  several  specific 
goals  for  increasing  the  volume  and  distribution  of  earth  science  data  and  products. 

NASA’s  fiscal  year  2001  performance  plan  still  lacks  an  explicit  discussion  of 
procedures  that  will  be  used  to  verify  and  validate  performance  data  and  does  not 
address  possible  data  limitation  issues  and  problems. 

Regarding  the  eight  new  fiscal  year  2001  performance  targets,  our  observations  are 
the  same  as  for  the  new  fiscal  year  2000  targets.  We  also  note  that  NASA’s  fiscal  year 
2001  performance  plan  still  relies  heavily  on  output  measures  for  this  key  outcome. 
We  believe  that  the  continued  use  of  output  measures  burdens  NASA  by  requiring  it 
to  continuously  demonstrate  the  linkages  between  program  efforts  and  results  and  to 
make  improvements  needed  to  strengthen  such  linkages. 
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Key  Agency  Outcome:  Deploy  and  Operate  the  International  Space  Station 
Safely  and  Cost-effectively 

Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Deploy  and  Operate  the  International  Space  Station  Safely  and  Cost-effectively 

Objective:  Deploy  and  operate  the  International  Space  Station  for  research, 
engineering,  and  exploration  activities. 

Targets: 

(1)  Deploy  and  activate  the  Russian-built  Functional  Cargo  Block  as  the  early 
propulsion  and  cargo  module. 

(Target  met.) 

(2)  Deploy  and  activate  the  first  U.S.  built  element,  Unity  (Node  1),  to  provide 
docking  locations  and  attach  ports. 

(Target  met.) 

(3)  Initiate  full-scale  Multi-Element  Integration  Testing  (MEIT)  for  elements  in  the 
first  four  launch  packages. 

(Target  met.) 

(4)  Deliver  the  U.S.  laboratory  module  to  the  launch  site  in  preparation  for  MEIT. 

(Target  met.) 

(5)  Conduct  the  physical  integration  of  the  Z1  Truss  launch  package  and  initiate 
MEIT. 

(Target  met.) 

(6)  Initiate  preparations  for  the  launch  of  the  first  Expedite  the  Processing  of 
Experiments  to  Space  Station  (EXPRESS)  rack  with  five  payloads  on  flight  7A.1. 
(Target  met.) 

(7) )  Initiate  preparations  for  the  launch  of  the  first  rack  of  the  Human  Research 
Facility  and  the  Window  Observation  Research  Facility  (WORF-1)  on  the  first 
utilization  flight. 

(Target  not  met.) 

GAO  Observations  on  NASA’s  Fiscal  Year  1999  Actual  Performance  for  the  Key 
Agency  Outcome  to  Deploy  and  Operate  the  International  Space  Station  Safely  and 
Cost-effectively 

NASA’s  fiscal  year  1999  performance  plan  and  report  do  not  include  performance 
objectives  and  targets  that  specifically  address  the  key  agency  outcome  of  deploying 
and  operating  the  International  Space  Station  (ISS)  safely  and  cost-effectively. 
However,  they  do  include  a  related  objective  of  deploying  and  operating  the  space 
station  for  research,  engineering,  and  exploration  activities.  Our  assessment  is  based 


Page  19 


GAO/NSIAD-00-192R  NASA’s  FY  1999  Performance  and  FY  2001  Performance  Plan 


Enclosure  I 


on  that  related  objective.  The  associated  performance  targets  adequately  indicate 
overall  progress  toward  the  related  objective  and  are  generally  objective  and 
measurable.  However,  all  of  the  performance  targets  appear  to  be  output  measures. 
(See  discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the  Earth 
system  for  NASA’s  comments  related  to  outcome  measures  and  our  general 
observation  of  this  issue.)  The  performance  information  clearly  articulates  the 
degree  to  which  the  annual  performance  targets  were  achieved  and  are  directly 
relevant  to  the  associated  performance  targets.  The  performance  report  includes  all 
targets  previously  established  in  the  fiscal  year  1999  performance  plan  for  this  related 
objective;  no  targets  were  excluded  or  revised.  NASA  met  all  of  its  targets  except  one 
for  this  related  objective. 

The  performance  report  includes  an  independent  evaluation  completed  in  February 
2000  by  the  NASA  Advisory  Council.  The  Council  identified  certain  significant  NASA 
efforts  that  NASA  did  not  include  in  its  performance  measurements.  The  evaluation 
indicated  that  space  station  goals  need  to  include  all  critical  events.  One  safety- 
related  effort  identified  by  the  Council  as  needing  performance  measures  was  the 
development  of  a  crew  return  vehicle  for  the  ISS.  The  Council  did  not  comment  on 
the  need  for  cost-control  performance  measures  for  the  space  station.  We  concur 
with  the  Council  on  NASA’s  need  for  safety-related  performance  measures  for  the 
space  station.  However,  we  also  believe  that  NASA  should  modify  the  related 
performance  objective  and  establish  targets  to  clearly  include  safety,  cost-control, 
and  risk  mitigation  measures. 

Discussion  of  data  credibility  is  limited  to  stating  that  each  of  the  program  and 
project  managers  are  fully  accountable  for  the  accuracy  of  the  performance 
information.  NASA’s  statement  does  not  provide  reasonable  assurance  that  the 
performance  information  is  credible.  (See  discussion  of  the  outcome  of  expanding 
scientific  knowledge  of  the  Earth  system  for  our  general  observation  of  this  issue.) 

Unmet  Fiscal  Year  1999  Performance  Goals  and  Measure  for  the  Key  Agency 
Outcome  to  Deploy  and  Operate  the  International  Space  Station  Safely  and  Cost- 
effectively 

(7)  Initiate  preparations  for  the  launch  of  the  first  rack  of  the  Human  Research 
Facility  and  the  Window  Observation  Research  Facility  (WORF-1)  on  the  first 
utilization  flight. 

GAO  Observations  on  NASA’s  Unmet  Fiscal  Year  1999  Performance  Goal  for  the  Key 
Agency  Outcome  to  Deploy  and  Operate  the  International  Space  Station  Safely  and 
Cost-effectivelv 

The  performance  assessment  indicates  that  target  7  was  only  partially  met.  We 
interpret  the  remanisfested  launch  as  indication  that  the  target  was  not  met  due  to 
unanticipated  delays,  and  we  view  this  as  a  reasonable  explanation  for 
nonachievement.  The  assessment  describes  clear  and  reasonable  actions  that  are 
being  taken  or  will  be  taken  to  complete  aspects  of  the  target  and  identifies 
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reasonable  time  frames  for  completion  of  specific  activities.  For  example,  the 
assessment  states  that  “while  continuing  in  its  early  design  and  development  phases, 
the  WORF-1  has  been  remanisfested  to  launch  on  Utilization  Flight-2.  It  has 
completed  its  system  requirements  review  and  preliminary  design  review.  During 
fiscal  year  2000,  it  will  complete  the  Critical  Design  Review  and  initiate 
manufacturing/assembly.  The  WORF-1  will  be  delivered  to  Kennedy  Space  Center  in 
mid-2001  to  support  a  launch  on  Utilization  Flight-2,  4  months  later  than  originally 
planned.” 

Fiscal  Year  2000  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Deploy  and  Operate  the  International  Space  Station  Safely  and  Cost-effectively 

The  objective  in  the  fiscal  year  2000  plan  was  revised  slightly  as  “Deploy  and  operate 
the  ISS  to  advance  scientific,  exploration,  engineering,  and  commercial  objectives.” 

In  our  view,  six  of  the  seven  fiscal  year  2000  performance  targets  are  new  for  this 
objective.  They  are  as  follows: 

■  Deploy  and  activate  the  U.S.  laboratory  module  to  provide  a  permanent  on-orbit 
capability. 

■  Deploy  and  activate  the  Canadian-built  Space  Station  Remote  Manipulator  System 
to  provide  an  ISS-based  remote  manipulating  capability  for  maintenance  and 
assembly. 

■  Deploy  and  activate  the  airlock  to  provide  an  ISS-based  extravehicular  activity 
capability. 

■  Deliver  to  orbit  the  first  of  three  Italian-built,  multipurpose  logistics  modules  to 
provide  a  reusable  capability  for  delivering  payload  and  systems  racks  to  orbit. 

■  Conduct  operations  with  a  three-person  human  presence  on  the  ISS. 

■  Complete  the  production  of  the  first  X-38  space  flight  test  article  in  preparation 
for  a  shuttle  test  flight  in  2001. 

In  our  view,  at  least  one  of  the  seven  targets  is  a  variation  of  a  fiscal  year  1999  target 
and  is  described  as  follows: 

■  Complete  preparations  for  the  initial  ISS  research  capability  through  the 
integration  of  the  first  rack  of  the  Human  Research  Facility,  five  EXPRESS  racks 
with  small  payload  research,  and  microgravity  science  glovebox  (variation  of 
fiscal  year  1999  target  7). 
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GAO  Observations  on  NASA’s  Fiscal  Year  2000  Planned  Performance  for  the  Key 
Agency  Outcome  to  Deploy  and  Operate  the  International  Space  Station  Safely  and 
Cost-effectively 

The  performance  report  does  not  provide  an  assessment  of  the  effect  of  fiscal  year 
1999  performance  on  estimated  performance  levels  for  fiscal  year  2000.  (See 
discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the  Earth  system  for 
our  general  observation  of  this  issue.) 

As  indicated,  six  performance  targets  for  fiscal  year  2000  are  new.  It  is  not  surprising 
that  performance  targets  change  each  fiscal  year,  since  launch  events,  which  are 
NASA’s  chosen  measures,  are  by  definition  unique.  However,  including  the  rationale 
for  newly  established  targets  in  annual  performance  plans  and  reports  would  clarify 
the  reasons  for  the  new  targets. 

Fiscal  Year  2001  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Deploy  and  Operate  the  International  Space  Station  Safely  and  Cost-effectively 

In  our  view,  all  six  performance  targets  for  fiscal  year  2001  are  new  for  this  related 
objective.  The  new  performance  targets  are  as  follows: 

■  Successfully  complete  the  majority  of  the  planned  development  schedules  and 
milestones  required  to  support  MEIT. 

■  Successfully  complete  the  majority  of  ISS-planned  on-orbit  activities,  such  as 
delivery  of  mass  to  orbit  and  enhanced  functionality. 

■  Successfully  complete  the  majority  of  combined  ISS-planned  operation  schedules 
and  milestones  as  represented  by  permanent  human  on-orbit  operations. 

■  Successfully  complete  the  majority  of  the  planned  research  activities  in  support  of 
initiation  of  on-orbit  research  opportunities. 

■  Successfully  complete  no  less  than  85  percent  of  the  planned  Russian  program 
assurance  schedules  and  milestones  required  for  the  development  of  the 
propulsion  module. 

■  Successfully  complete  no  less  than  75  percent  of  the  planned  crew  return 
capability  schedules.  Fiscal  year  2001  indicators  will  include  accomplishment  of 
program  schedule  milestones  for  phase  1  development  of  a  crew  return  vehicle 
that  could  provide  U.S.  crew  return  capability. 
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GAO  Observations  on  NASA’s  Fiscal  Year  2001  Planned  Performance  for  the  Key 
Agency  Outcome  to  Deploy  and  Operate  the  International  Space  Station  Safely  and 
Cost-effectively 

NASA  revised  its  approach  to  the  development  of  performance  targets  in  the  fiscal 
year  2001  performance  plan.  (See  discussion  of  the  outcome  of  expanding  scientific 
knowledge  of  the  Earth  system  for  our  general  observation  of  this  issue.) 

A  key  weakness  that  we  identified  in  NASA’s  fiscal  year  1999  and  2000  performance 
plans  related  to  this  objective  was  that  the  plans  did  not  provide  a  discussion  of 
procedures  for  verifying  and  validating  performance  data.  NASA’s  fiscal  year  2001 
plan  still  lacks  an  explicit  discussion  of  procedures  that  will  be  used  to  verify  and 
validate  performance  data  and  does  not  address  possible  data  limitation  issues  and 
problems.  (See  discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the 
Earth  system  for  our  general  observation  of  this  issue.) 

Regarding  the  6  new  fiscal  year  2001  performance  targets,  our  observations  are  the 
same  as  for  the  new  fiscal  year  2000  targets.  We  also  note  that  NASA’s  fiscal  year 
2001  performance  plan  still  relies  heavily  on  output  measures  for  this  related 
objective.  (See  discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the 
Earth  system  for  our  general  observation  of  this  issue.) 
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Key  Agency  Outcome:  Expand  the  Commercial  Development  of  Space 

Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  the  Commercial  Development  of  Space 

Objective:  Promote  investments  in  commercial  assets  as  pathfinders  in  ISS 
commercial  operations  and  reduce  the  cost  of  Space  Shuttle  operations  through 
privatization,  eventual  commercialization,  and  flying  payloads. 

Target: 

(1)  Complete  the  development  of  a  commercialization  plan  for  the  ISS  and  Space 
Shuttle  in  partnership  with  the  research  and  commercial  investment  communities 
and  define  and  recommend  policy  and  legislative  changes. 

(Target  not  met.) 

Objective:  Reduce  space  communications  and  operation  costs  through  privatization 
and  eventual  commercialization. 

Targets: 

(2)  Reduce  space  communication  operation  costs  30  to  35  percent  compared  to  the 
1996  budget  through  a  consolidated  space  communications  contract  to  meet 
established  budget  targets. 

(Target  met.) 

(3)  Develop  options  and  recommendations  to  commercialize  space  communications. 

(Target  not  met.) 

Objective:  Foster  consortia  of  industry,  academia,  and  government;  leverage  funding, 
resources,  and  expertise  to  identify  and  develop  space  commercial  opportunities. 

Targets: 

(4)  Increase  non-NASA  investment  (cash  and  in  kind)  in  space  research  from  $35 
million  in  fiscal  year  1996  to  at  least  $50  million  in  fiscal  year  1999,  a  40-percent 
increase. 

(Target  met.) 

(5)  Establish  a  new  food  technology  Commercial  Space  Center. 

(Target  met.) 
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Objective:  Revolutionize  space  launch  capabilities. 

Targets: 

(6)  Continue  the  X-33  vehicle  assembly  in  preparation  for  flight-testing. 

(Target  met.) 

(7)  Complete  vehicle  assembly  and  begin  flight-testing  of  the  X-34. 

(Target  not  met.) 

GAO  Observations  on  NASA’s  Fiscal  Year  1999  Actual  Performance  for  the  Key 
Agency  Outcome  to  Expand  the  Commercial  Development  of  Space 

The  associated  performance  objectives  and  targets  adequately  indicate  overall 
progress  toward  the  related  outcome  and  are  generally  objective  and  measurable. 
However,  almost  all  of  the  performance  targets  appear  to  be  output  measures.  (See 
discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the  Earth  system  for 
NASA’s  comments  related  to  outcome  measures  and  our  general  observation  of  this 
issue.)  Some  targets  include  quantifiable,  numerical  values  that  permit  a  comparison 
to  actual  performance  in  a  previous  year.  For  example,  target  2  proposes  reducing 
space  communication  operations  costs  30  to  35  percent  compared  to  the  1996  budget. 
Fiscal  year  1999  performance  results  for  this  target  indicate  that  space 
communication  costs  were  reduced  by  32  percent  compared  to  the  fiscal  year  1996 
budget. 

A  similar  example  is  target  4,  which  proposes  an  increase  in  non-NASA  investment 
(cash  and  in  kind)  in  space  research  from  $35  million  in  fiscal  year  1996  to  at  least 
$50  million  in  fiscal  year  1999.  Fiscal  year  1999  performance  results  for  this  target 
indicate  that  total  non-NASA  cash  and  in-kind  investments  totaled  $51.2  million  for 
fiscal  1999  compared  to  $35  million  in  fiscal  year  1996.  The  performance  report 
includes  all  performance  targets  previously  established  in  the  fiscal  year  1999 
performance  plan  for  this  outcome;  no  targets  were  excluded  or  revised.  The 
performance  information  clearly  articulates  the  degree  to  which  the  annual 
performance  targets  were  achieved  and  are  directly  relevant  to  the  associated 
performance  targets.  NASA  met  over  half  of  its  targets  for  this  key  outcome.  NASA 
met  all  of  its  targets  on  only  one  objective:  “Foster  consortia  of  industry,  academia, 
and  government;  leverage  funding,  resources,  and  expertise  to  identify  and  develop 
space  commercial  opportunities.”  It  did  not  meet  one  objective’s  target  and  met  a 
mix  of  targets  on  the  other  objectives. 

The  performance  report  includes  an  independent  evaluation  completed  in  February 
2000  by  the  NASA  Advisory  Council.  The  Council  commented  on  one  performance 
target  for  this  outcome,  target  6,  which  is  identified  as  having  been  achieved.  The 
Council  noted  that  while  assembly  of  the  X-33  vehicle  was  on  track  near  the  end  of 
fiscal  year  1999,  enormous  challenges  remain. 
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In  the  section  of  the  performance  report  that  relates  to  targets  1  through  5  of  this 
outcome,  discussion  of  data  credibility  is  limited  to  stating  that  each  of  the  program 
and  project  managers  are  fully  accountable  for  the  accuracy  of  the  performance 
information.  In  the  section  that  relates  to  targets  6  and  7,  additional  detail  indicates 
the  data  was  verified  by  senior  officials  at  the  field  centers  and  the  NASA  Program 
Management  Council  during  the  periodic  NASA  Enterprise  review  process.  However, 
the  detail  provided  does  not  provide  reasonable  assurance  that  the  performance 
information  is  credible.  (See  discussion  of  the  outcome  of  expanding  scientific 
knowledge  of  the  Earth  system  for  our  general  observation  of  this  issue.) 

Unmet  Fiscal  Year  1999  Performance  Goals  and  Measures  for  the  Key  Agency 
Outcome  to  Expand  the  Commercial  Development  of  Space 

(1)  Complete  the  development  of  a  commercialization  plan  for  the  ISS  and  the  Space 
Shuttle  in  partnership  with  the  research  and  commercial  investment  communities 
and  define  and  recommend  policy  and  legislative  changes. 

(3)  Develop  options  and  recommendations  to  commercialize  space  communications. 

(7)  Complete  vehicle  assembly  and  begin  flight-testing  of  the  X-34. 

GAO  Observations  on  NASA’s  Unmet  Fiscal  Year  1999  Performance  Goals  and 
Measures  for  the  Key  Agency  Outcome  to  Expand  the  Commercial  Development  of 
Space 

The  performance  assessments  indicate  that  target  1  was  partially  met  and  targets  3 
and  7  were  not  met.  The  assessment  for  target  1  does  not  explain  why  the  unmet  part 
of  the  target  that  was  not  met,  but  it  does  describe  clear  and  reasonable  actions  being 
taken  to  complete  aspects  of  the  target  and  identifies  reasonable  time  frames  for 
completion  of  the  target.  For  example,  the  assessment  states  that  the  development  of 
a  commercial  development  plan  for  the  ISS  was  completed,  while  the  development  of 
a  similar  plan  for  the  Space  Shuttle  has  been  delayed.  (The  reason  for  the  delay  was 
not  provided.)  The  assessment  indicates  that  George  Washington  University  is 
assisting  NASA  with  the  development  of  a  detailed  plan  for  the  Space  Shuttle  and  that 
completion  is  now  expected  in  March  2000.  The  assessment  provides  a  clear  and 
reasonable  explanation  for  why  target  7  was  not  met  but  does  not  indicate  why  target 
3  was  not  met.  According  to  the  assessment,  target  7  was  slowed  by  hardware 
delivery  problems  and  the  resolution  of  environmental  concerns  at  the  White  Sands 
Test  Facility.  The  assessments  for  both  targets  3  and  7  describe  clear  and  reasonable 
actions  that  will  be  taken  to  meet  the  targets  and  identify  reasonable  time  frames  for 
completing  the  actions. 

Fiscal  Year  2000  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  the  Commercial  Development  of  Space 

Fiscal  year  2000  objectives  1-3  for  this  outcome  have  been  substantially  revised. 


Page  26 


GAO/NSIAD-00-192R  NASA’s  FY  1999  Performance  and  FY  2001  Performance  Plan 


Enclosure  I 


In  comparing  the  revised  fiscal  year  2000  objectives  to  the  fiscal  year  1999  objectives, 
it  is  difficult  to  determine  from  the  rewording  whether  the  objectives  changed  in 
fiscal  year  2000.  The  revised  fiscal  year  2000  objectives  are  as  follows: 

■  Facilitate  access  to  space  for  commercial  researchers. 

*  Foster  commercial  participation  on  the  ISS. 

■  Meet  strategic  space  mission  operation  needs  while  reducing  costs  and  increasing 
standardization  and  interoperability. 

In  our  view,  9  of  the  12  fiscal  year  2000  targets  are  new  for  this  outcome.  They  are  as 
follows: 

■  Invest  25  percent  of  the  space  communications  technology  budget  by  fiscal  year 
2000  in  projects  that  could  enable  space  commercial  opportunities,  including 
leveraging  through  a  consortium  of  industry,  academia,  and  government. 

■  Foster  the  establishment  of  a  telemedicine  hub  in  Western  Europe.  NASA  and 
CNES  will  develop  an  international  telemedicine  program  to  incorporate  and 
connect  existing  medical  informatics  capabilities  into  a  user-friendly  commercial 
electronic  telemedicine  hub  and  apply  lessons  learned  to  human  space  flight. 

■  Utilize  at  least  30  percent  of  Space  Shuttle  and  ISS  fiscal  year  2000  capabilities  for 
commercial  investigations,  per  the  U.S.  Partner  Utilization  Plan. 

■  Promote  privatization  of  Space  Shuttle  operations  and  reduce  civil  service 
resource  requirements  for  operations  by  20  percent  (from  the  fiscal  year  1996  full¬ 
time  equivalent  levels)  in  fiscal  year  2000. 

■  Promote  privatization  and  commercialization  of  Space  Shuttle  payload  operations 
through  the  transition  of  payload  management  functions  (e.g.,  payload  integration 
managers  and  payload  officers)  by  fiscal  year  2000. 

■  Within  policy  limitations  and  appropriate  waivers,  pursue  the  commercial 
marketing  of  Space  Shuttle  payloads  by  working  to  allow  the  space  flight 
operations  contractor  to  target  two  reimbursable  flights,  one  in  fiscal  year  2001 
and  one  in  fiscal  year  2002. 

■  Increase  the  expenditures  for  commercial  services  to  10  percent  of  the  total  space 
communications  budget  by  fiscal  year  2000. 

■  Conduct  the  flight-testing  of  the  X-33  vehicle. 

■  Complete  small  payload-focused  technologies  and  select  concepts  to  support 
potential  decisions  on  the  flight  demonstration  of  a  reusable  first  stage. 

In  our  view,  at  least  3  of  the  12  targets  are  variations  of  fiscal  year  1999  targets.  They 
are  as  follows: 
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■  Establish  up  to  two  new  commercial  space  centers  (variation  of  fiscal  year  1999 
target  5) . 

■  Reduce  the  space  communications  budget  submission  for  fiscal  year  2000  30  to  35 
percent  from  the  fiscal  year  1996  congressional  budget  submission  (variation  of 
fiscal  year  1999  target  2). 

■  Complete  vehicle  assembly  and  begin  the  flight  test  of  the  second  X-34  vehicle 
(variation  of  fiscal  year  1999  target  7). 

GAO  Observations  on  NASA’s  Fiscal  Year  2000  Planned  Performance  for  the  Key 
Agency  Outcome  to  Expand  the  Commercial  Development  of  Space 

The  performance  report  does  not  provide  an  assessment  of  the  effect  of  fiscal  year 
1999  performance  on  estimated  performance  levels  for  fiscal  year  2000.  (See 
discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the  Earth  system  for 
our  general  observation  of  this  issue.) 

Of  the  12  performance  targets  for  fiscal  year  2000,  9  are  new.  In  previous  years, 

NASA  officials  stated  that  performance  targets  might  change  from  one  fiscal  year  to 
another  because  some  are  discrete  events  that  take  place  within  a  fiscal  year,  some 
change  with  the  phase  of  a  program,  and  others  are  expected  to  be  achieved  in  a 
specific  fiscal  year  with  no  carryover  to  a  later  fiscal  year.  Therefore,  many  of 
NASA’s  performance  targets  are  new  each  fiscal  year.  We  believe  this  is  reasonable  if 
NASA  establishes  performance  measures  that  cover  significant  efforts  and  critical 
management  issues  and  problems.  However,  including  the  rationale  for  newly 
established  targets  in  annual  performance  plans  and  reports  would  clarify  the  reasons 
for  the  new  targets. 

Fiscal  Year  2001  Performance  Goals  and  Measures  for  the  Key  Agency  Outcome  to 
Expand  the  Commercial  Development  of  Space 

The  fiscal  year  2000  objective  “Meet  strategic  space  mission  operations  needs  while 
reducing  costs  and  increasing  standardization  and  interoperability”  was  revised  in 
fiscal  year  2001  as  “Meet  sustained  space  operations  needs  while  reducing  cost.” 

The  fiscal  years  1999  and  2000  objective  “Revolutionize  space  launch  capabilities” 
was  removed,  and  a  new  objective  was  established  in  fiscal  year  2001.  The  new 
objective  is  “Reduce  the  payload  cost  to  low-earth  orbit  by  an  order  of  magnitude 
from  $10,000  to  $1,000  per  pound,  within  10  years,  and  by  an  additional  order  of 
magnitude  within  25  years.”  A  performance  target  was  not  established  for  the  X-33 
vehicle  as  was  in  fiscal  years  1999  and  2000.  The  performance  plan  indicated  that  a 
performance  indicator  for  the  X-33  could  not  be  identified  until  the  liquid  hydrogen 
tank  delamination  investigation  and  program  impact  assessment  were  complete. 

In  our  view,  four  of  the  five  performance  targets  in  the  fiscal  year  2001  plan  are  new 
for  this  outcome.  They  are  as  follows: 
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■  Establish  at  least  10  new,  active  industrial  partnerships  to  research  tomorrow’s 
space  products  and  improve  industrial  processes  through  NASA’s  Commercial 
Centers  and  find  opportunities  for  space  experiments. 

■  Foster  commercial  endeavors  by  reviewing  and/or  implementing  new  policies  and 
plans,  such  as  the  Space  Station  resource  pricing  policy  and  intellectual  property 
rights  policy.  Ensure  that  Space  Station  resources  allocated  to  commercial 
research  are  used  by  commercial  partners  to  develop  commercial  products  and 
improve  industrial  processes. 

■  Increase  the  percentage  of  the  space  operations  budget  allocated  to  acquisition  of 
communication  and  data  services  from  the  commercial  sector  to  15  percent  in 
fiscal  year  2001.  The  Space  Communications  Program  will  conduct  tasks  that 
enable  commercialization  and  will  minimize  investment  in  government 
infrastructure  for  which  commercial  alternatives  are  being  developed. 

■  Achieve  at  least  95  percent  of  planned  data  delivery  from  space  flight  missions  as 
documented  in  space,  ground,  deep  space,  and  NASA  integrated  service  networks 
performance  metrics  consistent  with  detailed  program  and  project  operation 
requirements  in  project  service  legal  agreements. 

In  our  view,  at  least  one  of  the  five  targets  is  a  variation  of  a  fiscal  year  1999  and  2000 
target. 

■  Complete  assembly  of  the  third  X-34  test  vehicle,  demonstrate  75  percent  of 
supporting  technology  developments,  and  complete  competitive  solicitations  for 
expanded  second-generation  reusable  launch  vehicle  efforts.  Indicators  for 
supporting  technology  development  include  both  flight  tests  and  ground  tests 
(variation  of  fiscal  year  1999  target  7  and  a  related  fiscal  year  2000  target  to 
“complete  vehicle  assembly  and  begin  the  flight  test  of  the  second  X-34  vehicle”). 

The  fiscal  year  2001  target  proposes  to  complete  assembly  of  the  third  X-34  test 
vehicle  but  also  identifies  additional  actions  that  will  demonstrate  75  percent  of 
supporting  technology  developments  and  complete  competitive  solicitations  for 
expanded  second-generation  reusable  launch  vehicle  efforts. 

GAO  Observations  on  NASA’s  Fiscal  Year  2001  Planned  Performance  for  the  Key 
Agency  Outcome  to  Expand  the  Commercial  Development  of  Space 

NASA  revised  its  approach  to  the  development  of  performance  targets  in  the  fiscal 
year  2001  performance  plan.  (See  discussion  of  the  outcome  of  expanding  scientific 
knowledge  of  the  Earth  system  for  our  general  observation  of  this  issue.) 

A  key  weakness  that  we  identified  in  NASA’s  fiscal  year  1999  and  2000  performance 
plans  related  to  this  outcome  was  that  the  plans  did  not  provide  a  discussion  of 
procedures  for  verifying  and  validating  performance  data.  NASA’s  fiscal  year  2001 
plan  still  lacks  an  explicit  discussion  of  procedures  that  will  be  used  to  verify  and 
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validate  performance  data  and  does  not  address  possible  data  limitation  issues  and 
problems.  (See  discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the 
Earth  system  for  our  general  observation  of  this  issue.) 

Regarding  the  four  new  fiscal  year  2001  performance  targets,  our  observations  are  the 
same  as  for  the  new  fiscal  year  2000  targets.  We  also  note  that  NASA’s  fiscal  year 
2001  performance  plan  still  relies  heavily  on  output  measures  for  this  key  outcome. 
(See  discussion  of  the  outcome  of  expanding  scientific  knowledge  of  the  Earth 
system  for  our  general  observation  of  this  issue.) 
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COMMENTS  FROM  THE  NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 


National  Aeronautics  and 
Space  Administration 

Office  of  the  Administrator 

Washington,  DC  20546-0001 


Mr.  Allen  Li 

Associate  Director  Defense  Acquisitions  Issues 
United  States  General  Accounting  Office 
Washington,  DC  20548 


Dear  Mr.  Li: 

NASA  appreciates  the  opportunity  to  comment  on  your  draft  report  on 
NASA's  FY  1999  Performance  Report  and  FY  2001  Performance  Plan.  NASA 
has  two  comments  with  the  report:  the  conclusion  that  the  revised  approach  in 
the  FY  2001  Performance  Plan  will  not  provide  improved  reporting  of  progress 
and  the  commentary  on  data  validation  and  verification. 

NASA  is  concerned  that  the  General  Accounting  Office  has  not  taken  the 
opportunity  to  assess  NASA’s  report  and  plans  in  the  context  of  the  goals  and 
missions  that  we  are  striving  to  achieve.  NASA  attempted  to  strike  a  balance, 
relating  current  expenditures  (and  outputs)  to  the  achievement  of  longer-term 
goals  and  objectives  (and  outcomes).  As  you  know,  in  a  research  and 
development  environment,  successes  are  hard  won  and  often  characterized  by 
steps  forward,  followed  by  interim  setbacks.  NASA  continues  to  explore 
approaches  that  put  both  successes  and  failures  into  a  complete  and  appropriate 
context.  As  Research  and  Development  outcomes  are  frequently  not  realized  for 
years  or  perhaps  decades,  NASA  feels  that  it  is  important  to  communicate 
annual  progress  towards  achievement  of  those  goals  to  our  stakeholders,  so  that 
they  might  understand  the  relationship  of  current  investments  to  the  realization  of 
those  longer  term  goals. 

In  restructuring  the  FY  2001  Performance  Plan,  NASA  took  another  step 
towards  demonstrating  the  relationship  of  annual  outputs  to  achievement  of 
objectives  such  as  “Improve  the  reliability  of  space  weather  forecasting."  Space 
weather  impacts  terrestrial  communications,  human  space  missions  and 
operations  of  spacecraft  engaged  in  exploration.  In  aggregating  a  series  of 
individual  programmatic  outputs,  NASA  intends  to  enable  the  demonstration  of 
trends  in  activities  such  as  the  collection  and  analysis  of  data,  which  will 
ultimately  allow  prediction  of  events  such  as  solar  flares  and  radiation  bursts.  The 
loss  of  data  from  a  single  operational  instrument  or  spacecraft  would  diminish  the 
knowledge  we  are  generating,  but  it  would  not  necessarily  render  us  unable  to 
ultimately  meet  the  objective  of  improved  space  weather  forecasting.  The  loss  of 
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data  from  multiple  instruments,  on  the  other  hand,  could  seriously  impact  our 
ability  to  improve  the  reliability  of  space  weather  forecasting. 

Our  other  major  concern  is  the  references  to  the  credibility  of  our  data. 
Unlike  many  other  Federal  Agencies,  the  collection  and  statistical  analysis  of 
data  provided  by  third  parties  are  not  a  primary  consideration  in  assessing 
NASA's  credibility  and  performance.  In  executing  NASA’s  mission  to  advance 
and  communicate  scientific  knowledge,  to  advance  human  exploration,  and  to 
research,  develop,  verify  and  transfer  advanced  technologies,  data  from  “other 
reliable  sources’’  are  not  available  to  verify  or  validate  reported  performance.  We 
rely  on  the  individuals  responsible  for  the  performance  to  validate  and  verify  the 
information  provided  to  us  for  Government  Performance  and  Results  Act 
compliance  and  we  also  rely  on  the  NASA  Advisory  Council  to  independently 
evaluate  our  reported  progress.  Your  brief  statements  on  data  verification  and 
validation  do  not  acknowledge  this  independent  evaluation  by  subject  matter 
experts,  although  you  do  repeat  the  NASA  Advisory  Council’s  remarks  in 
subsequent  sections  of  the  Report. 

NASA  has  no  major  issue  with  the  balance  of  the  findings.  Other  minor 
comments  and  responses  to  other  suggestions  including  data  validation 
concerns  are  provided  as  an  enclosure. 


Daniel  R.  Mulville 
Associate  Deputy  Administrator 


Enclosure 
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Data  Validation 

The  addition  of  detailed  statements  on  data  validation  for  each  of  the  targets 
would  significantly  increase  the  length  and  technical  detail  of  the  Performance 
Reports,  without  significantly  adding  value  for  most  readers. 

Internal  inconsistencies  in  tabular  comments 

Several  entries  note  that  NASA  did  not  comment  on  the  impact  of  FY1999 
performance  on  the  FY  2000  performance  targets  and  later  notes  that  the 
insertion  of  a  substantial  number  of  new  targets  is  appropriate,  e.g.  21  of  27 
Earth  Science  targets  were  "new."  With  the  introduction  of  the  new  targets, 
there  was  very  little  to  comment  on  in  the  FY  1999  report.  The  report  also 
indicated  plans  for  completion,  where  appropriate. 
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